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Abstract 

Reported in this article are initial results from of a longitudinal study to 
characterize the design cognition and cognitive design styles of high school 
students with and without pre-engineering course experience over a 2-year 
period, and to compare them with undergraduate engineering students. 

The research followed a verbal protocol analysis based on the function- 
behavior-structure (FBS) ontology, which employs a task-independent approach 
that is distinct from a task-based or an ad hoc approach. This approach to 
protocol analysis is applicable across any process-based view of designing and 
generates results based on a common comparative measure independent of the 
design task. 

In this article, Year 1 results are presented comparing only students in their 
junior year of high school who had formal pre-engineering course experience 
(experiment group) with those who did not have formal pre-engineering course 
experience (control group). Specifically, data collected from design sessions 
were analyzed for comparison of design issues and processes between 
experiment and control groups, respectively. Results from analysis of Year 1 
data did not reveal any significant differences between the experiment and 
control groups in engineering design cognition. Based on these results, one 
would conclude that students with pre-engineering course experience do not 
demonstrate a stronger focus on the process of producing design solutions than 
do students without such experience. Although analysis of demographic data 
from high school participants indicates some degree of common prior pre¬ 
engineering experiences, it did not provide a sufficient explanation for why no 
significant differences in engineering design thinking were found between these 
groups. The researchers anticipate that Year 2 data will indicate that as the pre¬ 
engineering students continue engaging in formal engineering design 
experiences during their final year of high school, some degree of difference in 
design cognition will be demonstrated. 

Keywords: design cognition; verbal protocol analysis; high school pre¬ 
engineering. 
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Background 

Engineering design used as an instructional strategy at the PK-12 level is 
increasingly being embraced as a core learning method and as a pedagogical tool 
for integrative STEM education (Kolodner, 2002, Wells, 2010). As a key 
stakeholder in this trend toward integration of engineering design in K-12 
STEM education curricula, it is critical that the elementary and secondary 
technology and engineering (T/E) education community understands the impact 
that such experiences have on student development of design practices. Few 
studies have examined the cognitive characteristics of K-12 students during T/E 
design-based learning (DBL) activities. Moreover, the way in which secondary 
students approach the engineering design process is not well understood (Katehi, 
Pearson, & Feder, 2009; Silk & Schunn, 2008) nor is whether that approach 
differs between students who have engaged in formal engineering experiences 
through pre-engineering course work and those who have not. Within the 
context of increasing opportunities for K-12 students to engage in both formal 
and informal T/E design activities, investigations regarding the extent to which 
such high school experiences contribute to a student’s capacity for design 
thinking (cognition) are needed. The intent of the research reported in this article 
was to characterize the design cognition of high school students and specifically 
to compare the design practices between high school students with and without 
formal pre-engineering design experiences. 

Though few would argue that the design literature in engineering education 
has been somewhat singularly focused on pedagogical issues, there is a growing 
body of literature from studies that seek to understand the characteristics of 
design thinking behavior from a cognitive viewpoint (Cross, 2004; Lawson, 
2004). Among these studies, protocol analysis is the research method of choice 
(Atman & Bursic, 1998; Dorst & Cross, 2001) for investigating design cognition 
and has been the basis for many of the more recent design cognition studies 
(Adams, Turns, & Atman, 2003; Atman et al., 2007; Christensen & Schunn, 
2007). The research study presented in this article followed a verbal protocol 
analysis based on the function-behavior-structure (FBS) ontology developed by 
Gero (1990) and its extension, the situated FBS (sFBS) ontology (Gero & 
Kannengiesser, 2004), as a design-based coding scheme. The FBS protocol 
analysis employs a task-independent approach, which is distinct from a task- 
based or an ad hoc approach. This approach to protocol analysis is applicable 
across any process-based view of designing and generates results based on a 
common comparative measure independent of the design challenge (task). In 
this way, the FBS protocol analysis addresses the underlying cognitive 
processes, as opposed to the standard behavior-based analysis, and therefore 
provides a uniform basis for comparisons between students with different 
educational preparation and backgrounds and from different educational 
environments (Jiang, Gero, &Yen, 2014; Williams, Gero, Lee, & Paretti, 2011). 
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Function-Behavior-Structure Verbal Protocol 

FBS Ontology 

The FBS ontology presents designing as the process of converting a set of 
functions into a set of design descriptions whereby those descriptions accurately 
convey an artifact capable of such functions (Gero, 1990). The design process is 
characterized in the FBS ontology (Figure 1) using three classes of ontological 
variables—function, behavior, and structure—as well as the external design 
requirements given the designer and a final description of the designed structure. 
Modeled in this way, function (F) is defined as the teleology of a designed 
object, and the behavior of that object is either what is expected (Be) from the 
structure or derived (Bs) from the structure. The structure (S) of an object 
represents individual components and the relationships among them. The 
external design requirements that the designer is given are designated by R, and 
the resultant set of design descriptions designated by D. These six ontological 
variables in the FBS model map onto design issues and serve as the basis for 
design cognition. 


6 



Be = expected behaviour -*- = transformation 

Bs = behaviour derived from structure -»-= comparison 

D = design description 
F = function 
R = requirement 
S = structure 

Figure 1 . FBS framework (Resource: Kannengiesser, Gero, Wells, Lammi, 
2015). 

A design description is the result of a designer having progressed through a 
set of eight distinct processes each of which reflects their movements (Numbers 
1-8, Figure 1) among the ontological variables. The first five processes reflect 
an implied linear sequence of movements that include formulation, synthesis, 
analysis, evaluation, and documentation. In formulation (1), requirements are 
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transformed into functions and functions into a set of expected behaviors; 
synthesis (2) results in a proposed structure to satisfy expected behaviors; 
analysis (3) of the proposed structure produces derived behaviors; in evaluation 
(4), both expected behavior and behavior derived from structure are 
concurrently assessed; and documentation (5) generates the design description. 
The iterative nature of designing is captured in the movement among three types 
of reformulation processes, which are also denoted numerically in Figure 1: 
Reformulation 1(6) is the reformulation of structure; Reformulation II (7) is a 
reformulation of expected behavior; and Reformulation III (8) is a reformulation 
of function. 

FBS Coding Scheme: Design Issues and Processes 

The coding scheme adhered to in this research is based on this FBS 
ontology whereby the ontological variables are translated into six design issues. 
These design issues are coded using the FBS ontology, as exemplified in the 
sample of participant utterances and associated codes seen in Table 1. The 
selected utterances were drawn from an engineering design session in which 
high school participants were asked to design a device that would assist elderly 
clients in opening a stuck double-hung window. Transformations between the 
six codes used to label the design issues reflected in participant utterances 
generate the eight distinct design processes (Table 2). 

Table 1 


FBS Coding Examples 


Design Issues 

Respective Utterance Example 

Design 

Requirements (R) 

"so they need help in trying to.... for the elderly to raise 
windows"; "it says a significant amount of force to raise 
and lower the windows..." 

Function (F) 

"but it'd have to be something that is really easy to 
twist."; "causes the window to expand on the frame" 

Behavior 

"that will increase mechanical advantage"; "that may 

Expected (Be) 

help the elderly lift or..."; 

Behavior from 
Structure (Bs) 

"so if they like pull the string it actually lifts it"; "so the 
longer this is the more mechanical advantage you'll have 
so the easier it will be" 

Structure (S) 

"So one thing I came up with is to cut a notch in the 
bottom frame of the window right there"; "and have the 
strings coming back down" 

Design 

Description (D) 

"let's draw a right side view of this thing to explain it 
okay I'll let you do that..." 


- 81 - 






Journal of Technology Education Vol. 2 7 No. 2, Spring 2016 

Unidirectional transformational movements are indicated by the ►" symbol, 
the symbol indicates transformational comparisons, and the numbers 
associated with each design issue correspond to those depicted in the FBS model 
(Figure 1). 

Table 2 

I BS Design Processes _ 


Progression _ Design Process _ Transformational Movement 


(1) 

Formulation 

R —> F, F —> Be 

(2) 

Synthesis 

Be—>S 

(3) 

Analysis 

S ->Bs 

(4) 

Evaluation 

Be <-> Bs 

(5) 

Documentation 

S —> D 

(6) 

Reformulation 1 

s->s 

(7) 

Reformulation II 

S ^Be 

(8) 

Reformulation III 

S —> F 


Method 

The research design followed a two-by-two factorial investigation across 
two exogenous variables, design experience and maturity, in which experience is 
formal pre-engineering coursework and maturity was the time between data 
collected fall of the junior and senior years of high school. The full scope of the 
research was to characterize the design cognition and cognitive design styles of 
high school pre-engineering students over a 2-year period and to compare them 
with undergraduate engineering students as well as high school students without 
such design experience. Presented in this article are Year 1 results comparing 
only the high school participants and only addressing the following hypothesis, 
which was one of six hypotheses posed in this study: High school pre¬ 
engineering students have a stronger focus than high school students with no 
design experience on the design process of synthesis (i.e., the process of 
producing solutions). 

Using purposeful selection, high school students in their junior year were 
assigned to experiment (those with formal pre-engineering course experience) 
and control (those without formal pre-engineering course experience) groups. In 
teams of two (dyads), students engaged in a predefined engineering design task 
in which they were to develop a design-only solution. A dyad configuration was 
used because it has been found to naturally promote authentic verbal interactions 
during collaborations on developing acceptable engineering design solutions 
(Kan & Gero, 2009; Purzer, Baker, Roberts, & Krause, 2008). 
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Participants 

Participants were drawn from a convenience sample of high school juniors 
attending one of three rural, mid-Atlantic high schools that offered the same 
ninth through twelfth grade Project Lead the Way (PLTW) pre-engineering 
course sequence. Student populations at each of the participating schools were 
of similar size. Two groups of participants, those with (experiment) and those 
without (control) formal PLTW pre-engineering course experience, were 
recruited from each high school, using a small monetary incentive. Prior PLTW 
course experience for the experiment group ranged from those enrolling in their 
first PLTW course at the start of their junior year to those with one full year of 
prior PLTW coursework. Within groups, students self-selected into dyads, 60% 
of which were mixed-gender. Of the 40 students participating in Year 1, the 
gender distribution within the experiment group was 64% male and 36% female, 
and for the control group, it was 65% male and 35% female. 

Procedures 

Participant recruitment was conducted using typical modes of school 
communication. Student demographic data (e.g., age, gender, pre-engineering 
course experience) were collected as students arrived at their session and before 
dyads engaged in the design task. The design task dyads addressed was that of 
designing a solution to assist physically impaired elderly nursing home residents 
with opening difficult-to-open, double-hung windows. Instructions for 
completing the design task were provided as well as basic information resources 
regarding the construction and operation of a double-hung window. Dyads were 
allowed 45 minutes to collaborate on their design task and instructed to include 
a detailed sketch of the final solution on a whiteboard. 

Data Collection and Protocol Analysis 

The following sequence of tasks presents the basic set of procedures used 
for data collection and protocol analysis. 

Video Capture. Each design task session was captured using two video 
cameras that were arranged at two distinct vantage points ensuring sufficient 
recording of dyad interactions and their development of a final design 
description (Figure 2). Additionally, the two video cameras safeguarded against 
potential technological issues or difficulties encountered by either device. The 
first camera directly captured the white board and dyads engaged in progressive 
design development and sketching of their solution, while second camera 
recorded a general view of the entire design session. Both members of a dyad 
were equipped with a high-sensitivity wireless microphone to ensure that quality 
audio was captured for successful transcription of student verbalizations into 
text. The resulting videos provide a time-stamped recording of the entire design 
session. 
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Figure 2. Participants sketching final design solution descriptions. 


Transcription. Video recordings of dyad design sessions were transcribed 
manually with individual utterances from each dyad member entered verbatim 
into alternating rows of a spreadsheet. Timestamps were inserted every three 
minutes to establish reference points throughout the entire video. This approach 
to transcription resulted in a written version of the verbalizations between 
participants with time stamps throughout. 

Segmentation and coding of text-based verbalizations. The method used 
to segment the text-based version of dyad verbalizations was conducted on the 
basis of the FBS coding that was previously described. This method involved 
concurrent analysis of a given transcript by independent coders. A total of six 
coders were involved with coding the 40 protocols. All coders participated in 
training using practice protocols until consistently achieving sufficient 
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intercoder reliability. Coders segmented and coded simultaneously, dividing the 
utterances until each individual segment contained a single code that reflected 
only one of the six possible design issues (Kan & Gero, 2007). The use of two 
independent coders ensured robustness and demonstrated an intercoder 
reliability ranging from 85% to 95%, which was consistent with prior research 
(Williams et al., 2011). 

Arbitration. After independent coders completed the segmentation and 
coding of a given transcript, they would meet to arbitrate—compare, discuss, 
and justify—the FBS codes that they assigned to each segment. When 
agreement of independently coded segments occurred, a final code was 
assigned. Segments that differed in assigned codes required coders to engage in 
arbitration to dispute the assigned coding and reach agreement on the design 
issue addressed. If coders were unable to agree on an arbitrated code, that 
segment was left uncoded and was highlighted for subsequent final arbitration 
between the lead researchers. The final arbitration resulted in a final protocol 
data set that was readied for use in statistical analyses. The number of segments 
typically generated from the final protocol for a 45-minute design session was 
between 200 and 700. Because there are six codes, this implies that, on average, 
each code would likely appear at least 33 times. This provides a statistically 
significant data set. Analyses of final arbitrated protocols were conducted using 
LINKODER (www.linkoder.com) to generate descriptive statistics and 
probability analyses of the FBS ontology. Data were analyzed to determine 
statistical differences in design issues and processes between the control and 
experiment groups. 


Results 

In this article, we report on the analyses of the first year data collected from 
design sessions of participating high school juniors. These data were analyzed 
for comparison of design issues and processes between experiment and control 
groups, pre-engineering (ENG) and nonengineering (NON), respectively. 

Design Issues 

A comparison of design issue distributions between ENG and control NON 
groups is illustrated in Figure 3, and descriptive statistics are presented in Table 
3. The percent occurrence reflects the average within group frequency of 
segments associated with each of the six design issues for both groups. The data 
indicate that both groups expended the majority of their cognitive efforts (~ 
40%) in discussions of the design structure (S), which is typical for most 
designers. Relatively similar total percent effort (-26-28) was expended on 
behavior from structure (Bs) and expected behavior (Be) combined (-14-18). 
Comparisons of control and experiment group data using a /-test (Table 3) 
revealed no significant differences among any of the design issues, although 
expected behavior approached it. Similarly, comparison of the total effort 
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expended in the problem versus the solution space (P-S Index; Jiang et al., 
2014) indicated that there were no significant differences. 

■ ENG ■ NON 


50 


45 



Requirement Function Eg?. Behztior Sir. Bekinot Structure Description 

Design Issues 

Figure 3. Percent occurrence of design issues: ENG vs. NON high school 
juniors. Error bars indicate standard deviation. 

Table 3 

Statistical Results of Design Issues (Entire Session): ENG vs. NON High School 


Juniors 

Design Issue_ t - value (%)p - value 

(R) Requirement -0.78 0.240 

(F) Function 1.05 0.153 

(B e ) Expected Behavior -1.7 0.053 

(B s ) Behavior from Structure 0.43 0.334 

(S) Structure -0.23 0.410 

(D) Description 1.09 0.145 

P-S Issue Index -1.2 0.123 


Design Processes 

The distribution of syntactic design processes was computed to discern 
differences in the cognitive effort expended between control and experiment 
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groups. Similar to computations of design issues, analytical comparisons of the 
eight syntactic design processes showed no statistically significant differences 
between ENG and NON groups (Table 4). No statistically significant differences 
were observed in the P-S Processes Index between these two groups. 

Table 4 


Statistical Results of Design Processes (Entire Session): ENG vs. NON High 
School Juniors 


Design Process 

t - value (%) 

p - value 

Formulation 

1.22 

0.118 

Synthesis 

-1.01 

0.163 

Analysis 

1.48 

0.077 

Evaluation 

-0.16 

0.436 

Documentation 

0.82 

0.211 

Reformulation 1 

-0.5 

0.311 

Reformulation 11 

-1.59 

0.064 

Reformulation III 

0.55 

0.295 

P-S Process Index 

-0.92 

0.183 


Percent occurrences for the eight design processes (Figure 4) indicate that 
roughly 30% of their cognitive effort was invested in Reformulation I (S > S) 
and between -17-21% on Analysis (B e oB s ). 
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■ ENG ■ NON 



Figure 4. Percent design processes occurrences: ENG vs. NON high school 
juniors. Error bars indicate standard deviation. 


Discussion and Conclusions 

Analysis of Year 1 data did not reveal any significant differences between 
the experiment (ENG) and control (NON) groups in engineering design 
cognition. Based on these results, the underlying hypothesis must be rejected: 
Pre-engineering students do not demonstrate a stronger focus on the process of 
producing design solutions. To further investigate this apparent lack of 
difference between ENG and NON groups, the following select demographic 
data related to prior T/E design experiences were collected: participation in (a) 
middle school technology education classes, (b) T/E clubs, (c) other T/E-related 
activities, and because of the rural school settings, (d) farm-related activities. 
Analysis of these data indicated that of the ENG students, 59% had previously 
participated in middle school technology education classes, 14% were or had 
been involved in T/E clubs, 30% engaged in other T/E-related activities, and 
30% had T/E-related farm experiences. In each of these demographic categories, 
students in the NON group had significantly less additional formal or informal 
T/E-related experiences, 33%, 5%, 17%, and 0% respectively. It is evident from 
these demographic data that students in the ENG group had far more formal and 
informal T/E-related experiences. 

Although demographic data indicates some degree of common prior pre¬ 
engineering experiences, it does not provide sufficient explanation for finding 
no significant differences in engineering design thinking between these groups. 
Other influences such as curricular and pedagogical factors must therefore be 
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considered. Project Lead the Way (PLTW) program documents present entry- 
level course outlines that do not specifically target design thinking as a learning 
goal (https://www.pltw.org/our-programs/engineering). This is equally the case 
for the curriculum used by the middle school technology education programs at 
participating schools. The initial PLTW course that all pre-engineering 
participants engaged in was Introduction to Engineering Design (IED). A review 
of the detailed IED curriculum outline indicates that instructional units give 
attention to teaching the following set of practices and steps in the design 
process: technical sketching and drawing skills, modeling skills, geometry of 
design, documentation, and completion of a prescribed design project using 
computer-aided design (CAD) software. Authentic open-ended design 
challenges are not integral to the learning experience provided to students in this 
entry-level pre-engineering course. In light of this, it suggests that the 
pedagogical preparation provided to educators delivering the earlier courses in 
PLTW might not be adequate for intentionally incorporating or promoting 
design thinking as part of pre-engineering experiences. 

Year 2 data of this longitudinal study are currently being collected. The 
researchers anticipate that as the pre-engineering students continue their 
engagement in engineering design experiences during their final year of high 
school, differences in design cognition will be demonstrated to some degree. 
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